AO— AX 06  726  LOGICON  INC  SAN  DIEGO  CA  TACTICAL  AND  TRAINING  SYSTE—ETC  F/G  5/9 

MEASUREMENT  OF  STUDENT  ACHIEVEMENT  FOR  AIR  INTERCEPT  CONTROLLER— ETC (U) 
AUG  61  R  M  ANDERS*  M  P  GANNIS*  R  HALLEY  N61339-78-C-0162 

UNCLASSIFIED  0641-AOO7  NL 


Cl 


NAVAL  TRAINING  EQUIPMENT  CFNTFR 
ORLANDO  F LOR* DA  32*];, 


1 

I 


81  11  0  3  011 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  QW  TN|»  MQt  (9*m  Omm 

REPORT  DOCUMENTATION  PAGE 


NAVTRAEQUIPCEN  78-C-0182-5 

i  TtTLC  (mu r  _ _  _ _ 

y  Measurement  of  Student  Achievement  for  ) 

Air  Intercept  Controller  - — — —  -  ■■■'■ ^ 

Prototype  Training  System  ^  ) 

^  ii  ■  |— - i 

y  AUTHOR^ 

Ronald  MyAnders/^iichael  P./Gannis,^4— " 
l  BobinfHalley  amd  Elaine  C./Regelson  J  V 

t  rerforming  organization  name  anoTSoSeis^™™ 

Logicon,  Inc.,  Tactical  &  Training  Systems  Div. 

Post  Office  Box  80158 

San  Diego,  California  92138 

11  CONTROLLING  OFFICE  NAME  AMO  ADORSSS 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


Naval  Training  Equipment  Center 
Orlando,  Florida  32813 


NUMBER  OE  RAGES 

_  16 _ 

MONITORING  agency  name  *  AOOftliSTTf  dffimtm »l  It 55  Control  ting  Otfic!)  I  IS.  SECURITY  CLASS,  (of  HXi  roportT 

Unclassified 


AftSlRlCATlON/OOWNGRAOlNG 

dULE 


I  nr  OISTRIEUTION  STATEMENT  (of  thla  Kaport) 


Approved  for  public  release;  distribution  unlimited. 


17.  DISTRIBUTION  STATEMENT  (of  tha  abatroct  an  to  rod  In  Block  20.  If  different  from  Kaport) 


10.  SURRLEMENTARY  notes 


19-  KEY  WOROS  (Contlnuo  on  reverae  at  da  if  nacaaaory  and  idpntify  by  block  mm  bar) 

VTAG  (Voice  Technology  Advisory  Group) 

AIC  (Air  Intercept  Controller) 

Automated  Instruction  Methods 

^  - .  — 

This  report  discusses  the  Measurement  of  Student  Achievement  which, 
utilized  in  the  experimental  prototype  Air  Intercept  Controller 
Training  System,  presents  formats  for  automated  measurement  reporting. 


inn 


ip 


DD  1  JAN *73  1473  EDITION  OF  1  NOV  €%  IS  obsolete 


_ UNCLASSIFIED _ 

SECURITY  CLASSIFICATION  OF  THIS  RAGE  r**#"  Data  Entered) 


NAVTRAEQUIPCEN  78-C-0182-5 


MEASUREMENT  OF  STUDENT  ACHIEVEMENT 

INTRODUCTION 

The  purpose  of  this  report  is  to  present  the  achievement  outcomes  of 
student  involvement  with  the  Prototype  Air  Intercept  Controller  Training 
System  (hereafter  referred  to  as  ACE).  The  report  will  present  a  brief  intro¬ 
duction  to  the  ACE  system's  instructional  approaoh,  will  delineate  the  methodology 
by  whioh  the  achievement  data  was  collected,  will  present  the  available  results, 
and  will  discuss  areas  in  which  criterion  achievement  was  not  aohleved. 

The  ACE  system  was  designed  to  provide  a  research  tool  for  exercising 
the  use  of  several  new  "risk"  technologies  in  a  military  training  environment. 
These  technologies  (automated  speech,  automated  instructor,  and  vldeodiso) 
are  all  being  implemented  at  a  "state-of-the-art"  level  and  are  being  carefully 
perused  to  determine  potential  benefits  of  their  use  iQ  military  training 
systems.  The  ACE  system  design  included  visible  criterion  measures  of  student 
achievement  within  the  curriculum  to  allow  a  Judgment  of  student  aoblevament 
levels.  ACE  utilizes  a  mastery  approaoh  to  instruction.  That  is  to  say, 
the  student  is  not  allowed  to  proceed  in  the  curriculum  until  he  has  mastered 
the  current  topic  at  the  criterion  level. 


Student  achievement  on  the  system  is  tested  in  three  ways.  Initially, 
the  learner  Is  given  a  pretest  to  determine  whether  he  has  all  the  required 
skills  and  knowledges  which  make  him  eligible  to  be  a  candidate  student  in 
the  AIC  school.  When  he  has  passed  the  pretest  or  has  been  given  remediatlve 
instruction  to  enable  him  to  develop  the  skills  tested  on  the  pretest,  the 
student  enters  the  curriculum.  From  that  point  on  the  student  is  constantly 
encountering  progress  tests. 

Progress  tests  on  ACE  take  four  forms:  (1)  knowledge  tests,  (2)  simple 
skill  tests,  (3)  commented  practices,  and  (4)  free  practices.  The  ACE  system 
uses  knowledge  tests  and  skill  checks  to  confirm  initial  acquisition  of  the 
knowledges  and  skills  required  for  the  AIC  job.  Once  the  learner  acquires 
the  "know  how"  in  an  instructional  environment,  he  is  given  the  chance  to 
practice  this  competence  in  a  simulated  job  environment.  There  the  abilities 
are  tested  using  specified  behaviors  as  indicators  of  success.  During  practices 
the  ACE  system  uses  84  performance  measurement  variables  (PMVs)  to  measure 
the  learner's  achievement  on  these  behaviors. 


A  failing  score  on  the  knowledge  and  skills  tests  can,  on  critical  skills, 
require  that  the  student  review  some  of  the  instruction  previously  encoun¬ 
tered  or  can,  on  less  important  skills  or  knowledge,  simply  result  in  feedback 
giving  the  correot  answer. 


On  the  commented  and  free  practices,  a  failing  score  means  that  the 
learner  will  have  to  repeat  the  practice.  A  certain,  predefined  number  of 
failures  will  cause  the  learner  to  be  automatically  sent  to  remediatlve  instruc¬ 
tion  or,  in  more  extreme  oases,  be  referred  to  the  human  instructor  for  further  — 
instruction. 
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The  last  time  the  student  is  asked  to  perfora  each  of  the  measured  behaviors 
in  a  free  practice  segment  his  behavior  is  considered  to  be  the  posttest 
on  that  skill.  The  segments  in  which  these  post tests  occur  are  called  mastery 
segments  (see  Appendix  A,  ’Mastery  Segments").  Passing  these  serpents  constitutes 
successful  learner  achievement  within  this  system. 

METHODOLOGY 

The  major  measure  of  achievement  to  be  monitored  was  how  well  the  learner 
did  on  each  of  the  84  PMVs.  Also  to  be  monitored  was  how  long  it  took  eaeb 
learner  to  go  through  the  currioulum  and  what  sorts  of  speclflo  problems 
he  had.  Specific  free  practice  segments  were  designated  as  mastery  segments 
for  each  of  the  PMVs.  The  designated  segments  were  generally  those  that 
tested  the  PMV  for  the  final  time  in  the  curriculum. 

Five  students  completed  the  currioulum  during  the  test  period.  Bach 
of  the  students  was  monitored  oonoerning  how  well  they  did  on  each  of  the 
PMVs  during  the  free  praotice  segments  as  a  measure  of  their  aohievmaent. 
The  nans  of  the  student,  the  date  the  test  was  administered,  the  identification 
of  the  particular  test,  and  the  student's  deficiencies  are  all  available 
for  inspection  and  were  used  in  preparation  of  the  results  reported  below. 

TEST  RESULTS 

The  overall  results  of  the  PMV  use  by  the  first  five  students  are  presented 
in  Appendix  B.  What  these  overall  results  show  is  a  very  limited  sueoess 
for  the  students  on  the  84  system  PMVs,  although  there  is  some  suggestion 
from  instructor  comments  and  system  observation  that  student  scores  have 
improved  as  the  system  has  been  updated  and  as  the  instructors  have  beoome 
used  to  working  with  ACE. 

ACHIEVEMENT  OF  CRITERIA  STANDARDS.  During  the  demonstration  test  period 
only  one  practice  (CP  2.04)  was  passed  by  any  of  the  five  students.  Therefore, 
a  further  disousslon  of  learner  achievement  on  criteria  standards  appears 
irrelevant. 

The  students  have,  however,  passed  individual  PMVs  during  these  practice 
sessions,  and  a  look  at  groups  of  these  PMVs  suggests  how  well  the  training 
is  working  in  different  areas  of  instruction.  For  purposes  of  this  report, 
the  PMV  data  have  been  scrutinized  and  the  following  results  are  presented 
as  observations. 

ANALYSIS  OF  PERFORMANCE  ERRORS.  The  training  curriculum  for  ACE  can  be  first 
divided  into  tactical  and  aircrew  setup  training  environments.  The  tactical 
environment  can  be  further  divided  into  stationing,  bogey  engagement  and 
runout,  bogey  jinks,  bogey  splits,  strangers,  rendezvous,  fades,  Naval  Taetioal 
Data  System  (NTDS)  program  failures,  emergencies,  and  general  skills.  The 
aircrew  setup  training  can  also  be  subdivided.  These  areas  are  stationing, 
separation,  state,  Intercept,  and  general  skills.  Observations  are  presented 
for  the  taotical  area  first  and  then  for  the  aircrew  setup  training  area. 
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All  but  on*  of  the  stationing  skills  PKVs  are  based  on  speech  (aider standing. 
The  learners  have  passed  these  PKVs  (2,  3,  4,  14,  15,  19,  20,  28,  29)  no 
average  of  half  the  time.  Only  PMV  19  (AIC  *roger"s  airborne  for  control 
message)  has  never  been  passed.  There  does  not  appear  to  be  a  real  improvement 
correlated  with  training  time  on  these  skills. 

The  highest  level  of  pass  on  any  of  the  bogey  lnteroept  skills  PMVs 
(5,  6,  7,  6,  9,  22,  23,  24,  25,  26,  27,  30)  is  about  50  percent  with  aost 
of  the  FNVs  showing  a  0  to  25  percent  passing  occurrence.  The  level  of  passing 
seeas  to  remain  fairly  constant  over  the  course  of  the  ourrloulua.  The  PMV 
measuring  the  bearing  and  range  call  to  the  bogey  shows  a  definite  degradation 
aa  additional  skills  are  added. 

Jinks  skills  PMVs  (31  ,  32,  33)  have  shown  a  combination  of  problems. 
Only  one  of  these  have  been  passed  and  that  one  only  once.  The  oauae  of 
at  least  soae  of  this  seeas  to  be  in  the  PMVs  rather  than  the  students'  skills. 

Splits  skills  PMVs  (34,  35,  82,  83,  84,  85,  86,  87)  have  been  passed 
very  few  tiaes.  Part  of  the  reason  for  this  may  be  that  it  is  a  very  complicated 
area  and  that,  in  fact,  the  learners  have  not  mastered  the  skills.  A  contributing 
factor  may  be  that  the  computer  simulation  presentation  of  information  to 
the  student  has  been  incorrect.  These  two  factors,  combined  with  at  least 
one  PMV  with  a  measurement  error,  probably  have  contributed  to  the  low  scores 
in  this  area. 

Stranger  associated  skills  are  somewhat  of  an  anomaly.  None  of  these 
skills  have  ever  been  passed.  It  haa  been  suggested  that  this  is  a  combination 
of  inoorrect  measurement  by  the  PMVs,  slightly  confusing  scenario  setups, 
and  a  digit  recognition  problem  discussed  later  in  this  report. 

The  success  of  the  learners  on  the  rendezvous  skill  PMVs  (40  ,  41,  42, 
43,  44,  45,  46)  has  been  very  low.  These  results,  however,  do  not  match 
the  instructors'  opinion  of  how  well  the  learners  are  mastering  these  skills. 
Their  impression  has  been  that  the  learners  ouster  these  skills  fairly  well. 

The  problem  is  with  the  PMVs  and  digit  recognition. 

The  two  PMVs  (47,  48)  which  measure  the  learner's  ability  to  detect 
and  call  radar  fades  have  been  passed  an  average  25-50  percent  of  the  time. 
No  reason  has  been  suggested  for  this  low  achievement  level. 

Student  response  to  NTDS  program  failure  is  measured  using  three  PMVs 
(49,  50,  5D.  Student  scores  on  these  PMVs  have  been  unusually  high  for 
speech  related  tasks.  PMVs  49  and  51  were  passed  67  percent  of  the  time 
and  PMV  50  was  passed  33  percent  of  the  time. 

Three  PMVs  (52,  53 »  54)  are  used  to  measure  student  response  to  airorew 
emergencies.  None  of  these  PMVs  has  ever  been  triggered,  due  to  an  apparent 
problem  in  scenario  setup,  and  there  is  no  data  on  this  instructional  area. 

The  general  area  is  comprised  of  PMVs  (1,  10,  11,  12,  13,  16,  17,  18) 
that  are  measured  during  more  than  one  part  of  the  AlC's  tasks.  Most  of 
these  are  console  aotion  tasks  and  show  high  scores.  PMV  1  (tracking  the 
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Coffloat  Air  Patrol  (CAP)  synbol)  shows  a  uniform  high  passing  oocurrenoe  through 
Level  7.  Then,  in  Level  8  when  the  learner  is  asked  to  traok  two  symbols 
at  the  same  time,  the  scores  drop.  As  Level  8  goes  on,  the  scores  reoover 
to  a  high  level. 

The  PMV  (78)  measuring  the  AIC  student's  ability  to  get  the  airorews 
to  the  operational  area  was  never  triggered.  Therefore,  there  is  no  achievement 
data. 


The  skills  associated  with  separating  aircraft  for  intercept  in  the 
training  environment  are  measured  by  seven  PMVs  (  57*  58,  59*  68,  79*  80, 
81).  The  scores  on  these  PMVs  are  uniformly  medium  to  high.  The  exceptions 
are  57  and  79  whiob  were  never  triggered  and  81  which  was  never  passed. 

Obtaining  and  updating  NTDS  fuel  states  is  covered  by  three  PMVs  (70, 
71  ,  72) .  The  scores  for  this  area  are  fairly  high.  They  appear  to  be  higher 
than  for  the  same  skills  in  the  tactical  environment.  It  has  been  suggested 
that  this  is  because  the  AIC  students  are  not  as  busy  in  the  training  environment 
when  they  get  the  state  reports  as  they  are  when  they  ask  for  fuel  states 
in  the  tactical  environment. 

The  Intercept  skills  peculiar  to  the  airorew  setup  environment  are  measured 
by  six  PMVs  (62,  63,  64,  65,  66,  67).  The  results  of  this  area  are  confusing 
with  similar  skills  showing  a  range  of  0  to  80  percent  passing.  The  low 
scores  have  been  attributed  to  PMV  problems  and  the  instructors  have  suggested 
that  the  indicated  ability  of  the  learners  is  about  medium  high  on  these 
skills. 

The  last  area  in  aircrew  training  behavior  measurement  is  the  general 
category.  The  PMVs  (55,  56,  73,  74,  75,  76,  77)  show,  as  with  the  general 
PMVs  in  the  tactical  environment,  a  fairly  high  passing  level.  The  exceptions 
to  this  are  PMV  55  which  was  never  triggered  and  PMVs  69  and  75  which  were 
never  passed.  These  3  PMVs  are  presently  assumed  to  have  design  problems. 

INTERPRETATION  OF  TEST  RESULTS 

The  lack  of  achievement  data  provided  by  the  ACE  system  during  the  training 
effectiveness  evaluation  period  appears  to  be  primarily  the  result  of  the 
research  and  development  approach  required  for  implementing  the  several  "risk" 
technologies  wbioh  ACE  encompasses,  rather  than  the  ability  of  the  system 
to  train. 

As  with  most  projects  of  a  researoh  and  development  nature,  the  imple¬ 
mentation  is  an  evolutionary  period  during  which  the  relationships  between 
the  experimental  factors  of  the  system  muat  be  iteratively  revised.  On  the 
ACE  system  the  several  technologies  being  experimentally  applied  generated 
a  larger  set  of  tuning  requirements  than  anticipated.  Thus,  time  and  effort 
originally  allocated  for  tuning  the  Performance  Measurement  Variables  (PMVs), 
courseware,  and  speech  systems  were  expended  in  getting  the  major  system 
components  to  funotlon  correotly.  That  left  very  little  time  to  attend  to 
PMVs,  courseware,  and  speech  system  fine  tuning.  As  a  result,  when  the  first 
set  of  learners  went  through  the  system  there  were  still  a  set  of  unresolved 
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issues  which  restricted  the  ability  of  the  students  to  exercise  the  various 
system  capabilities.  Major  contributors  to  these  restrictions  were  the  unpolished 
state  of  the  PMVs,  soenario  setups,  speech  understanding,  digit  recognition, 
systoa  stability,  and  the  videodisc  player.  These  constraints  will  be  discussed 
in  sons  detail  later  in  this  report. 

The  outoome  of  the  restrictions,  however,  was  that  the  students  were 
not  able  to  proceed  through  the  systea  curriculum  as  anticipated.  As  a  direct 
result,  there  is  very  little  achievement  data  available,  and  that  data  is 
not  useful  as  a  valid  measure  of  student  achievement.  The  rest  of  this  section 
will  discuss  the  factors  which  contributed  to  the  limited  achievement  success 
and  will  suggest  system  revisions  whiob  might  remove  those  constraints. 

PERFORMANCE  MEASUREMENT  VARIABLES  AND  AUTOMATED  SPEECH.  This  probably  has 
been  the  major  constraint  on  the  system  in  terms  of  achievement.  There  are 
several  areas  which  still  require  test  and  revision  before  the  automated 
performance  measurement  system  will  work  as  envisioned. 

There  are  84  PMVs.  At  the  time  the  first  learners  went  on  the  system, 
none  of  these  had  been  tested  in  position.  As  a  result  of  the  PMV  analysis, 
it  was  determined  that  there  was  one  anticipated  problem  and  several  unanti¬ 
cipated  problems. 

It  was  anticipated  that,  with  84  PMVs,  some  PMVs  might  not  work  as  designed. 
Indeed,  there  are  problems  in  triggering  of  the  PMVs  ?.nd  measurement  of  behaviors 
which  have  beoome  evident  through  the  recent  student  use.  Improper  IMV  triggering 
has  resulted  in  several  of  the  PMVs  having  never  been  used  and  others  continuing 
to  measure  behaviors  after  they  should  have  been  turned  off.  As  they  presently 
work,  the  PMVs  with  measurement  deficiencies  have  led  to  the  learner  being 
scored  incorrectly  for  correct  behaviors. 

The  solution  for  these  two  problems  is  fairly  simple.  Practice  scenarios 
will  have  to  be  run  which  exercise  each  of  the  PMVs.  The  PMVs  will  be  studied 
individually,  deficiencies  and  problems  will  be  noted,  and  revisions  will 
be  made.  The  PMVs  should  then  be  checked  again  to  insure  that  the  revision 
has  resolved  all  the  PMV  errors. 

It  was  not  anticipated  that  there  would  be  problems  with  timing,  but 
two  such  areas  have  been  identified  in  the  analysis. 

The  first  timing  problem  was  caused  by  the  speech  understanding  system. 
By  the  time  a  phrase  goes  through  the  speech  recognition  procedures  and  then 
through  the  speech  understanding  processes,  up  to  two  seconds  can  elapse. 
By  the  time  the  system  requests  a  message  be  sent  baok  to  the  learner  and 
that  message  is  sent,  another  four  seconds  oan  go  by.  When  giving  a  student 
a  limited  amount  of  time  to  do  certain  tasks,  the  additional  six  seoond  delay 
can  nearly  insure  failure.  This  problem  has  been  partially  solved  by  loosening 
of  the  time  frames  for  success  on  the  PMVs,  but  nothing  has  been  done  at 
this  point  about  the  time  the  student  has  to  spend  waiting  fcr  a  system  response. 
There  is  no  obvious  solution  to  this  problem  short  of  redesigning  the  system. 
Designing  higher  speed  access  between  speech  and  the  rest  of  the  system  might 
speed  up  the  response  time. 
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The  other  area  where  timing  was  a  factor  had  to  do  with  the  amount  of  time 
it  actually  takes  a  person  to  perform  a  behavior.  In  an  analysis  of  ideal  AIC 
behavior,  the  AIC  School  staff  and  the  Logicon  training  analysts  defined  very 
specific  time  frames  for  the  performance  of  each  successive  behavior.  These 
were  reviewed  and  approved  and  were  used  as  the  basis  for  FMV  design.  However, 
software  design  failed  to  consider  the  Human  Factor  of  beginning  measurement 
at  the  point  when  the  event  was  recognized  by  the  trainee.  The  result  of  this 
is  that  the  learners  will  fall  behind  on  one  task  (failing  it  as  a  result)  and, 
in  a  domino-like  fashion,  fall  behind  on  successive  task  after  successive  task 
(failing  them  as  a  result). 

A  solution  to  this  problem  is  for  the  AIC  school  staff  and  knowledgeable 
system  personnel  to  design  and  implement  a  more  realistic  set  of  PMV  start 
events,  stop  events,  and  time  frames.  This  could  be  researched  and  implemented 
at  the  same  time  the  PMVs  are  being  redesigned. 

A  third  unanticipated  factor  was  that  the  speech  understanding  system 
might  have  trouble  handling  the  digit  sets  involved  in  vector,  bearing  and 
range,  track  and  ground  speed,  altitude,  and  state  messages.  Part  of  the 
problem  is  associated  with  the  speech  recognition  unit  (Nippon  Electric  Corpora¬ 
tion  DP- 100).  This  unit's  digit  recognition  was  less  adequate  than  anti¬ 
cipated.  Another  part  of  the  problem  was  with  the  speech  understanding  system 
(SO S)  ,  which  was  a  complicated  portion  of  the  ACE  system.  At  times  the  less 
than  perfect  speech  recognition  data  was  rendered  more  illegible  for  perfor¬ 
mance  measurement  by  the  SUS  processes.  The  result  was  that  most  of  the 
PMVs  which  measured  skills  involving  transmission  of  digits  were  rarely  passed, 
if  at  all.  The  complicated  bearing  and  range  and  track  and  ground  speed 
calls  suffered  most  from  this  deficiency.  Since  the  digit  related  PMVs  are 
very  common,  it  became  nearly  impossible  for  a  learner  to  pass  any  given 
practice  segment. 

Solutions  have  been  suggested  for  resolving  the  digit  recognition  con¬ 
straint.  The  most  likely  answer  at  this  point  is,  when  the  learner  reaches 
the  end  of  a  segment,  to  notify  him  that  the  system  is  having  problems  recognizing 
his  digit  transmissions,  but  to  take  off  no  points  for  this  mlsrecognition. 
This  will  allow  the  learner  to  be  aware  of  transmission  problems  that  can 
cause  undesirable  responses  from  the  simulated  personnel  models  and  yet  not 
fail.  The  system  will  still  deduct  points  for  transmissions  which  are  too 
early,  too  late,  or  in  the  wrong  general  direction  (e.g. ,  starboard,  port). 

SCENARIO  SETUPS.  The  way  the  practice  scenarios  were  set  up  caused  some 
minor  training  problems.  Although  most  of  the  scenarios  appear  to  be  adequate 
for  accomplishing  the  designated  training  objectives,  a  few  scenarios  either 
do  not  allow  the  learner  time  to  do  all  the  behaviors  that  are  required  or 
do  not  supply  the  required  interaction  between  the  system  and  the  student 
(e.g.,  no  jink  occurs  when  jinks  responses  are  being  tested;  in  some  oases 
the  PMV  fails  to  trigger  within  its  Mastery  Segment). 

The  solution  to  the  scenario  setup  problem  is  very  muob  like  the  PMV 
measurement  solution.  Given  a  little  time  to  exercise  the  system,  experienced 
system  personnel  could  make  the  required  changes  in  a  relatively  short  time. 
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SYSTEM  STABILITY.  The  ACE  system  Is  a  complicated  network  of  programs  supporting 
a  large  number  of  technologies.  As  might  be  expected  early  during  the  imple¬ 
mentation  period  of  a  research  and  development  oriented  prototype  system* 
the  system  can  still  encounter  situations  that  cause  it  to  bang  up  or  orash. 
This  has  happened,  with  decreasing  regularity,  on  the  ACE  system,  causing 
the  loss  of  some  training  time  that  might  have  enhanced  learner  achievement. 
The  solution  to  this  problem  is  primarily  one  that  Involves  time.  Historically 
with  ACE,  as  each  constraint  appears  and  is  diagnosed,  it  is  remedied.  It 
is  reasonable  to  believe  that,  with  sufficient  resources,  such  corrections 
can  oontlnue. 

A  major  factor  in  terms  of  system  stability  has  been  the  tendency  of 
the  computer  hardware  to  come  down  "hard."  Early  in  the  test  period  there 
were  recurring  problems  in  keeping  the  computer  equipment  up,  which  cost 
a  great  deal  of  training  time.  The  solution  is  to  contine  with  a  high  level 
preventive  maintenance  program. 

VIDEODISC  PLAYER.  The  process  of  producing  videodisc  materials  is  a  fairly 
simple  one.  The  technologies  associated  with  videotape  and  film  production 
are  well  known.  However,  the  videodisc  player  technologies  still  are  relatively 
young  and  rough  around  the  edges. 

The  videodisc  player  that  was  used  in  the  ACE  system  was  a  laser  optical 
type.  It  uses  a  laser  beam  to  "read”  the  videodisc  encoded  information. 
When  the  laser  goes  out  of  alignment,  as  it  did  frequently  on  the  ACE  system, 
the  address  and  picture  information  are  no  longer  available.  This  makes 
it  impossible  to  address  single  frames,  of  which  there  were  over  800  on  the 
ACE  system  dlso.  As  a  result,  the  learner  no  longer  has  any  of  the  visual 
materials  available  during  training. 

One  short  term  solution  is  to  have  a  backup  player  available  for  use 
while  the  primary  player  is  being  realigned.  A  longer  range  solution  is 
to  try  other  types  of  players  or  to  wait  until  the  technology  is  more  advanced. 

CONCLUSIONS  AND  RECOMMENDATIONS 

The  report  up  until  this  point  has  concentrated  on  the  limited  achievement 
data  generated  by  the  ACE  system,  the  apparent  reasons  contributing  to  these 
limitations,  and  possible  solutions  for  resolving  present  system  restrictions. 
However,  this  picture  is  not  totally  comprehensive  in  terms  of  the  ability 
of  the  ACE  system  to  train.  Indeed,  the  feeling  at  the  AIC  school  and  at 
Logloon  is  that  the  system  holds  great  promise  for  teaching  AIC  skills. 
The  AIC  school  staff  is  continuing  to  put  additional  students  onto  the  ACE 
system,  despite  its  current  constraints. 

There  are  presently  restrictions  on  the  ACE  system's  ability  to  function 
as  conceived.  Most  of  these  problems  appear  to  be  amenable  to  fairly  simple 
"fixes"  (see  "Problems  and  Solutions"  earlier  in  this  report). 

In  order  to  accumulate  any  reliable  data  on  learner  achievement,  it 
is  going  to  be  necessary  to  revise  some  PMVs  to  provide  more  adequate  measurement 
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and  to  relieve  the  digit  recognition  problem.  Although  there  are  other  con¬ 
straints,  these  two  obanges  should  alleviate  most  of  the  major  restrictions 
and  allow  the  system  to  be  exercised  much  mere  closely  to  its  capabilities. 
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APPENDIX  A 
MASTERY  SEGMENTS 

The  ACT  ;  ystcn  measures  student  performance  in  pr  actice  segments  through 
«.i  use  of  HA  perforcdnco  Measurement  variables  (PMVs).  Er.sh  of  the  skills 
.<r.ng  measured  is  carefully  taught  during  the  curriculum  and  then  tested 
for  i  x'inal  tire  in  what  is  termed  a  "Mastery  Segment"  for  that  PMV.  The 
tc  lowing  list  presents  the  8A  PMVs  and  the  number  of  its  mastery  segment. 

TABLE  A1 .  PMV  MASTERY  SEGMENTS 


MASTERY 

samcE? 

EMY-MME 

1 

7 .70 

Maintain  CAP  Symbol  In  Vicinity  Of  CAP  Video 

7.06 

Engage  CAP  To  Station 

7.06 

Transmit  (Initial)  Station  Bearing  And  Range 

04 

7  .06 

Transmit  (Continuing)  Bearing  and  Range  Of  Station 

oe 

7.20 

Engage  CAP  To  Bogey 

06 

7.20 

Vector  CAP  To  Bogey 

07 

7.20 

Transmit  Initial  Bogey  Bearing  And  Range 

o  >■ 

7.20 

Transmit  Initial  Bogey  Track  And  Ground  Speed 

0* 

7.20 

Transmit  Continuing  Bogey  Bearing  And  Range 

1 0 

6.10 

Ensure  TEC  Communication  Switches  Ar  Correct 

11 

7.20 

Ensure  TEC  Control  Panel  Switches  Arc  Correct 

V* 

7.20 

Range  Scale  And  Offset 

r-> 

7.06 

Enter  CAP  Symbol,  PIF,  and  Station  Altitude 

u 

7.06 

Airborne  For  Control 

7.06 

Ruth,  This  Is  C/S... 

it 

7.06 

Update  CAP  Symbol 

17 

7.06 

Ask  CAP  For  State 

i  > 

7.06 

Update  NTDS  With  CAP  State  (non- training  environment) 

:• 

7.06 

Notify  SWC  Of  Control 

pi 

7.06 

On  Station 

[Deleted] 

?' 

7.20 

Transmit  Bogey  Composition  And  Altitude 

?' 

7.20 

Place  Bogey  On  Sequence  List 

7.20 

Respond  To  Judy  Or  Tally  Ho 

*'» 

7.20 

Lost  Contact 

2i 

7.20 

Contact 

"{ 

7.06 

Disengage  CAP  From  Bogey  At  Breakaway 

p' 

7.06 

Re-Engage  CAP  To  Station  After  Break  .w^y 

p  ( 

7.06 

Vector  CAF  To  Station  After  Breakaway 

7.06 

Report  Results  Of  Engagement 

7.20 

Transmit  Jink  Call 

'j  ' 

'.20 

Transmit  Vector  To  Counter  Jin 

o 
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VAJJ.i.  At .  PtSV  MASTERY  SEGMENTS  -  continued 


MASTERY 

SEGMENT 


m  NAME 


7.20 
7.06 
7.06 
7.06 
7.06 
7.06 
7.06 
6.13 
6.13 
’  .13 

6.1'i 

6.13 
6.17 

6.1'j 

7.20 

7.30 

7.14 
7.14 
7.14 
7 .20 
7.20 
7.20 
8.10 

6. 17 
3.1? 
3 . 1 0 

8.17 


8.10 

8.10 

8.10 

8.17 

8.17 

6.17 

8.17 

8.10 

3.17 

8.1 7 

8.17 
'..in 
6.10 
8.10 

3.17 
8.10 
3. 10 
3.17 


Transmit  Updated  Bogey  Track 

Transmit  Bogey  Splitting 

Transmit  New  Bogey  Composition,  Altitude 

Detect  And  Report  Strangers 

Call  Stranger  Bearing  And  Range 

Transmit  Stranger's  Track  And  Ang< Is 

Stranger  Opening 

Transmit  Vectors  For  Rendezvous 

Attain  Correct  Lateral  Separation 

Transmit  To  The  MAC  Bearing  And  Range  To  The  CAP 

Transmit  MAC'S  Altitude  To  CAP  For  Rendezvous 

Measure  Rendezvous  Flight  Path 

Measure  Rendezvous  Separation 

Transmit  To  The  CAP  Bearing  And  Range  To  The  MAC 

Fighter  In  The  Dark 

Bogey  In  The  Dark 

Transmitting  NTDS  Down  Message 

Initial  Bearing  And  Range  Transmit,  I'.TDS  Down 

Conti n.  Bearing  And  Range  Transmit,  KTDS  Down 

Establishing  Comm,  After  Alar.--  (Beeper  On  Guard) 

Reporting  CAP  Emergency  To  SWC 

Check  Emergency  Plot  Position 

Select  32  Mile  Range  Scale  For  Se  Ups 

Keep  Aircraft  In  The  Area 

Breakaway 

Disengage  Pseudo  Bogey  From  Point-In-Space  (B) 
Disengage  CAP  From  Point-In-Space  (A) 

[Deleted] 

[Deleted] 

Engage  Pseudo  Bogey  to  PP0I 
Engage  CAP  To  ?P0I 
Disengage  CAP  From  PF0I 

Establish  Initial  And  Final  Intercept  Condition 

Vector  CAP  To  3ogey  In  Training 

Engage  CAP  To  Pseudo  Bogey  In  Training 

Measure  Setup  Separation 

Establish  Lost  Communications 

Update  NTDS-State 

Request  Pseudo  Bogey  State  (Training) 

Request  CAP  State  (Training) 

Enter  CAP  Symbols  And  PIF 

Range  Scale  md  Offset  (Training  Environment) 

Update  Turn  late 

Pseudo  Bogey  Symbol  Update 

Upaate  Pseudo  Fogey  Symbol 

Direct  CAP  To  Center  Of  Area 

Engage  Pseudo  Bogey  To  Point 


! 

1 

1 

! 

1 

i  nv 

-rul* 

TAM. 

MASTERY 

tni 
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:  At.  HIV  MASTERY  SEGMENTS  -  continued 

EMY-M1E 

i 

| 

80 

H.17 

F.n  Age  CAP  To  Point 

.1  * 
j 

81 

8.10 

Detach  Wingman 

i 

j 

8.'1 

7.08 

Disengage  CAP  From  Split  At  Breakaway 

j 

83 

7.06 

Disengage  CAP  From  Bogey  After  Break  Engage  Alert 

\ 

8k 

7.06 

Engage  CAP  To  Split 

\ 

65 

7.06 

Vector  CAP  To  Split 

i 

KA 

7.06 

Transmit  Initial  Split  Bearing  And  Range 

i 

1  ■; 

7.06 

Transmit  Continuing  Split  Bearing  And  Range 
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APPENDIX  B 

t’M.V  ACHIEVEMENT  SUMMARY 

Five  students  went  through  the  ACE  system  curriculum  during  the  demonstration 
period.  The  following  table  summarizes  how  well  the  students  did  on  each 
of  the  8M  PKVs.  The  data  is  presented  in  terms  of  number  of  passes/number 
of  attempts  for  each  of  the  Free  Practice  (FP)  exercises  contained  in  the 
ACF,  curriculum. 
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